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PREFACE 



This handbook has been prepared to guide evaluators in the Office 



of Research of the Pittsburgh Public Schools in the systematic planning 
of their activities under the Discrepancy Evaluation Model, ^ 



As the first of two volumes, the handbook has two principal objec- 



tives: first, to orient the new evaluator to the philosophy of evaluation 



inherent in the discrepancy model; and second, to outline in detail the 



often routine administrative processes necessary to conduct day to day 



activities in the first two stages of evaluation. 



The handbook necessarily makes repeated references to specific 



regulations, documents, and procedures of the Pittsburgh Public Schools, 



Nevertheless, it has been written so that it can be easily adapted tQ any 



school system which subscribes to the concept of evaluation set forth in 



the discrepancy model. For it is the model* s underlying logic, applicable 



to educational programs in general, that has led to the detailed evaluation 



procedure and duties described in the present volume. 



Leonard E* Glassner 
Office of Research 



Malcolm Provus 
Director of Research 



1. Discrepancy Evaluation Model, 1969 (Pittsburgh Public 
Schools, 1969 ). 
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Section I 



ORIENTATION 



A. BECOME FAMILIAR WITH THE ORGANIZATION OF THE 
SCHOOL SYSTEM 

A minimal working knowledge of the Pittsburgh Public Schools as 
an urban school system is a prerequisite foif the evaluator before he 
associates himself with any specific program. He can obtain this basi< 
information frorfi several sources. 



Legal regulations and official policy are detailed in the Rules of the 



Board of Public Education of the School District of Pittsburgh, to which 
revisions are made as changes of policy occur. This publication con- 
tains a simplified organization chart of the school system. 

Three quasi- official manuals which contain useful information are 
the Handbook of Personnel Procedures. Handbook for Teachers, and 

for Principal^. Although out of print and somewhat outdated, 
a booklet entitled Your Virst Year in the Pittsburgh Public Schools has 
many useful obser vation\s about staff functions and duties and contains 



insights into inter per sona\^ relationships within the system. A copy of 
each of these publications i^s available in the Office of Research library. 

As part of the orientation program for new evaluators, staff mem- 
bers of the Office of Research will provide additional material related 
to the evaluation effort. These staff members are a continuing source of 
chgr ound information, and should be consulted whenever the 




evaluator feels that their experience will be helpful in clarifying specific 
problems, 

B. - BECOME FAMILIAR WITH EVALUATION RATIONALE 
AND PROCEDURES 

The evaluator must be able to explain the overall rationale and gen- 
eral procedures for evaluation in the Pittsburgh Public Schools before 
accepting responsibility for any program, Reading the Discrepancy 
E^/al uai-ion Model, 19^9 arid participating in an orientation workshop on 
evaluation are necessary first steps in acquiring this ability. Each of 
these activities is discussed below, 

1„ READ DISCREPANCY MODEL 

The Discrepancy Evaluation Model is the basic document for under- 
standing the theory and procedures of evaluation research in the Pitts- 
burgh Public ochools. The theory assumes that the continuous reporting 
of observed differences between a program staff's expectations for a 
program and the field reality will lead decision makers to effect desirable 
changes. This reporting follows well-defined stages which include devel- 

t-'' 

opment of a precise program design by staff at all levels, "careful study 
. . .of their program operations, a detailed analysis of program inputs 
and processes, and the verfication that programs are in fact operating 
as people believe them to be operating, 

The above quotation is from the Discrepancy Ev a luation Model 
which details the prescribed procedures for carrying a program through 
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the several stages of evaluation. The evaluator will need to read this 
document thoroughly before participating in the orientation workshop. 
Throughout this manual, specific references will be made to the Discrep- 
auc y Evaluation Model and subsidiary publications. 

2. PARTICIPATE IN EVALUATION IN-SERVICE WORKSHOP 
Whenever a new group of evaluators joins the staff, the Coordinator 
of Evaluation will conduct an orientation workshop. This workshop is 
designed to provide a comprehensive introduction to evaluation philosophy 
and technique. Before the workshop is convened, the new evaluator will 
have learned about the basic organization and operation of the Pittsburgh 

f 

Public Schools and will have read the Discrepancy Evaluation Model. He 

will then be able to participate in the practicum of evaluation activity 
described below« 

When an evaluator is employed at a time when no formal workshop 

IS imminent, the Coordinator of Evaluation will arrange an informal 

orientation for him, covering the same subjects considered in the 
workshop. 

a. EVALUATION TECHNIQUE 

The first part of the orientation workshop will acquaint the evalua- 
tor with general techniques and procedures for implementing the model. 
Although each workshop will have its own specific agenda, the evaluator 
will take part in such typical activities as those described below . 
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b. VIEW FILM, "DISCREPANCY feVALUATION MODEL" 

This 25-minute sound, film illustrales through animation and docu- 
mentary tecliniques how the discrepancy model guides educational and 

f 

evaluative staffs in designing and monitoring programs. Considerable 
attention is given to the design as the standard for comparison in mea- 
suring a program*s effectiveness. The film specifically considers the 
stages and cycles through which a developing program progresses. It 

'i 

shows how frequent feedback can produce Improvement in instructional 
and service-oriented programs. The interaction between evaluators 

I 

and program staff throughout the life of a p^'ogram is emphasized. 

I 

' I 

c. GENERATE A SAMPLE PR0GRAM\ DESIGN 

A major activity in the workshop will be ‘the construction of a design 
for a specific progi*am according to model spdjcifications, A frequently 
used technique is to provide the evaluator with an early description of a 
program obtained before the model was developied. After identifying gaps 
in the description, he can proceed to build an improved design, A logi- 
cal follow-up is participation in a mock panel meeting convened to judge 
the adequacy of the design. The most recent official design of the program 
can serve as a standard against which the evaluator can judge his own 
proficiency as a program designer. 
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d, PLAN STRATEGY AND PROCEDURE FOR DESIGN MEETING 
OF SAMPLE PROGRAM 

It is impGrative that careful planning precede the design meeting, 



since its product will be the standard by which the program is continually 
evaluated. The first set of plans concerns the routine administrative 
details. These require painstalcing attention, in order to assure the 
smooth functioning of the meeting. The purely clerical and logistical 
matters will be handled by the secretarial staff and the administrative 



assistant on instructions from the evaluator. The content of the meet** 
ing itself, however, is the complete responsibility of the evaluator. 
Such items as the following must typically be considered in planning a 
design meeting: 

1, Selecting an appropriate date for the meeting 

2, Reserving an adequate meeting place 

3, Determining and notifying the participants of the meeting 

4, Structuring the discussion groups 

5, Arranging for substitute teachers, if necessary 

6, Preparing the questions for the discussion groups from 
which content for the program design will be obtained 

7, Selecting arid briefing discussion leaders 

8, Arranging for recording the proceedings of the meeting. 
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PREPARE SAMPLE FLOWCHART OF PROGRAM PROCESS 



The ability to construct a flowchart of the program process is a 
valuable skill. It enables the evaluator to see botla the relationship of 
major (terminal) and enabling objectives for a program and the sequence 
of activities and procedures by which the objectives are realized. The 

r 

standardized symbols and procedures for flowcharting are presented 
in ‘'Flowcharting Techniques, " a pamphlet published by the IBM Corpora- 
tion, and available from the librarian, A less formal approach will be 
found in the annual report for 1968 of the Standard Speech Developncient 
Program, The evaluator may wish to experiment with other variations 
of flowcharting techniques for his sample flowchart. 



f. DESIGN SAMPLE STAGE II INSTRUMENT 

There are few standardized instruments for collecting the kinds of 
data required in btage II evaluation. Therefore, most such instruments 
will need to be locally constructed. By examining the instrument file 
in the evaluation office and referring to past reports of specific programs, 
the evaluator will find many examples of questionnaires, interview 
schedules, and other instruments suitable for Stage II monitoring. It 
should be noted that a rationale must be provided for every instrument 
and section thereof. The Coordinator of E"\^luation and a consultant, if 
necessary, will assist the evaluator in instrument design. Nevertheless*, 
as part of the orientation workshop, the evaluator should try his hand at 
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designing a sample instrument to measure some aspect of program 



operation. He will also prepare a rationale along the lines of those he 



has examined to accompany the instrument. 



g. WRITE SAMPLE CYCLE REPORT 



The preceding workshop activities have been designed to prepare 



the evaluator for the writing of a cycle report. With this background, 



he should begin the present assignment by examining the latest design 



of the program he intends to work with in this exercise. He should then 



read the most recent cycle report for the program and note its findings. 
This will lead him to the next step- -the determination of the probable 



sources of discrepancy between design and implementations for inves- 



tigation in the forthcoming cycle of evaluation. He can then prepare a 



series of questions designed to elicit the information he needs for his 



cycle report. The findings related in the sample cycle report will neces- 



sarily be fictional. At the same time, they should be consistent with actual 



program operation, and should include plausible alternative courses of action. 



3. RECEIVE HUMAN RELATIONS TRAINING 



At this point it will be clear that the evaluator must control many 
technical skills as he carries out his duties. No matter how greaf his 



technical proficiency, however, without the ability to interact effectively 
with people at all levels of program responsibility, he will fail at his 



task. By its very nature,, the Discrepancy Evaluation Model often requires 
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the reporting of negative findings. If program staffs lack confidence in 



the evaluator or feel threatened by evaluation activities, they are unlikely 



to take remedial action to improve their programs. For tliis reason, the 



second part of the orientation workshop will be devoted to interpersonal 



relationships. The three objectives for this aspect of the workshop are: 



1. To enable the evaluator to identify the circumstances when 
interaction with program staff is necessary or advisable 



2. To provide the evaluator with the human relations techniques 
suitable for working with program staff at several levels 
under various conditions 



3, To equip the evaluator with the skills needed to assess his 
own sensitivity in dealing with others 



During the orientation, realization of these objectives will be ap- 



proached through role playing, T-grouping, or other methods of sensi- 



tivity training, as well as through selected reading. Directly following 



the orientation period, new evaluators will observe their experienced 



colleagues in a variety of situations. 
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Section II 

STAGE I EVALUATION 

A. PLANNING EVALUATION ACTIVITY 

The first section of this category is developed in two parts. This 

division reflects the fact that a beginning evaluator will proceed in a 
somewhat different way when he accepts responsibility for a new program 
than he will when he accepts responsibility for a program that has been 
in operation for some time. 

* 

1. ACCEPT RESPONSIBILITY FOR EVALUATION OF NEW 
PROGRAM 

The first task for an evaluator of a program not yet in operation is 
to become thoroughly familiar with its initial planning. The Coordinator 
of Evaluation will brief the evaluator on the conditions that led adminis- 
trators to propose the program. 

Where a formal proposal has been submitted, as in the case of 
federally and state funded programs, a rationale will usually be stated 
in general terms. In addition, the proposal will provide some clues as 
to overall objectives, target populations, and process variables that 
decision makers had in mind when they conceived the program. The 
formal proposal will also provide some indication of contemplated evalua- 
tion activity. In ideal situations, the evaluator will participate in for- 
mulating the evaluation section of the proposal. It should be remembered, 



however, that in most cases significant departures must be made from 
the original specifications of the proposal to meet changing needs and 
the reality of actual installation. 

Where a formal proposal is not available, the Coordinator of Evalua- 
tion will direct the evaluator to the best sources. for obtaining the pre- 
liminary information described above, 

a, READ RELEVANT LITERATURE 

The rationale for any justifiable educational program rests on theo- 
retical assumptions with which the evaluator should become conversant. 
The Office of Research maintains a library to aid the evaluator in famil- 
iarizing hir^self with the literature relevant to his specific program. It 
subscribes to m.any periodicals in various disciplines and purchases books 
covering a wide range of topics. It regularly receives reports and ab- 
stracts of evaluation and research being done in other organizations. The 
librarian is the custodian for these resources, 

][n his reading the evaluator will often find reports of programs sim- 
ilar to the one for which he is responsible. He may find that certain 
evaluation aspects of a program in another city are generalizable to his 
own. Familiarity with the relevant literature can alert, the evaluator to 
pitfalls he should avoid in his own work, and can also save considerable 
time by suggesting suitable instruments and procedures for data collection. 
He should not overlook the fact that knowledge of the literature is also a 
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great asset in communicating with program staff as he seeks to gain 
their confidence, 

b, ESTABLISH THE PROGRAM FILE 

The master files for all programs in the Office of Research are 
systematically organized and maintained by the librarian. The evalua- 
tor should make sure, however, that the librarian is aware of any new 
program so that proper files may be established. These files should 
contain the basic documents and correspondence pertaining to each 
program from its conception to its termination. The evaluator is free 
to borrow any materials from the files in accordance with established 

f 

procedures, and will be held responsible for their security while they 
are in his possession. As a program develops, the evaluator will pro- 
vide the’ librarian with all important records for cataloguing and filing. 
This includes tapes of meetings as well as written records. 

It is a good idea for the evaluator to keep his own files to document 
the progress of each of his programs. Besides copies of official docu- 
ments and letters, including memoranda, these files will ordinarily 
contain excerpts of appropriate literature and notes of meetings and 
other contacts with program staff. This is not to say that the evaluator 
should try to have a duplicate of everything in the master program file, 
nor that his own files should not be periodically culled. 
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c. PREPARE CYCLE PLAN FOR OBTAINING INITIAL PROGRAM 
DESIGN. 



The evaluator will remember the emphasis placed upon the design 



meeting during the orientation sessions. Following the procedures out- 



lined in section d on page 5, he should prepare a cycle plan and accom- 



panying timeline .for holding a design meeting for his new program. The 



plan in draft form will be submitted to the Coordinator of Evaluation for 



approval and possible revision. Certain items may need to be discussed 



with the program manager before the plan is put into final form. 



d. MAKE INITIAL CONTACT WITH PROGRAM MANAGER 



The initial contact between the program manager and the evaluator 



can be made as soon as the Coordinator of Evaluation has approved the 
tentative plan for the program design meeting. This contact serves two 



purposes; First, it provides an excellent opportunity for the Coordinator 



of Evaluation to introduce the evaluator to the program manager and set 



the stage for a productive relationship. Second, it enables the program 



manager and the evaluator to work together on a crucial activity, each 



within his own sphere of responsibility, early in a program*s history. 



(1) DISCUSS OVERALL EVALUATION PROCEDURE WITH 
PROGRAM MANAGER 



It is imperative that the program manager accept the evaluator as 



a knowledgeable and influential professional with a legitimate interest 
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in the program. At this first meeting, the evaluator should create the 
impression that he will serve as an objective monitor and reporter of 
program developments in the context of the discrepancy model. It 
' ' • should be made clear at this meeting that the evaluator is not a source 
for program content which remains the responsibility of the program 
staff., 

Before discussing the specifics of the design meeting, the key pro- 
visions and philosophy of program evaluation under the discrepancy model 
should be presented. Among the items to be mentioned are the stages of 
evaluation, feedback procedures, and the necessity for periodic data 
collection. 







(2) DISCUSS PLANS. TO OBTAIN INITIAL PROGRAM DESIGN 



Once he understands the function of the program design, the pro- 



gram manager can offer valuable help in setting up the proposed design 



meeting. He should be invited to suggest alternate meeting dates and 



sites and the names of personnel not included in the original list of par- 



ticipants, Although the specific questions which the evaluator has drawn 



up for generating the design will not be mentioned at this preliminary 



meeting, the evaluator can legitimately ask the program manager to 



suggest several topics that might properly be brought up in the discussion 



groups. Those topics that will lead to a more precise design can be 



phrased as questions and considered in the design meeting. 



1 
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e. MAKE SPECIFIC PLANS TO OBTAIN FIRST PROGRAM 
DESIGN 

in preparing for the design meeting the evaluator should be guided 
by the information contained in section d on page 5 above (PLAN 
STRATEGY AND PROCEDURE FOR DESIGN MEETING OF SAMPLE 
program)* If it is possible, the Coordinator of Evaluation will arrange 
for the evaluator to observe the planning activities for a design meeting 
of another program as well as the meeting itself. He should also partici- 
pate in the evaluation staff's critique of the meeting in order to acquire 
additional insights into the rationale and strategy of the design meeting 
and to benefit from suggestions for continuous improvement of the pro- 
cess of obtaining a program design. 

(1) PREPARE QUESTIONS FOR DISCUSSION LEADERS 
The basic structure for the design meeting is a series of questions 
relating to the input, process, and output variables of a particular pro- 
gram, Each of these categories can be subdivided into sections appro- 
priate for the program being designed. Figure 3 of the Discrepancy 
Evaluation Model (p. 17) contains the general areas in the three basic 
categories which should be used to generate the questions concerning 
the specific variables of a program. Each question should be worded 
so that it will elicit the most precise answer possible. Where feasible, 
a question should seek to determine the exact standard which the program 
staff considers a measurement of attainment for a given variable. 
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To obtain a reasonably complete design, a fairly large number of 
questions will have to be asked at the design meeting. Each question 
should serve to produce a particular segment of the total program design 
This is true of the questions suggested by the program manager as well 
as those proposed by the evaluator and other members of the evaluation 
staff. A careful study should be made of questions used in other design 
meetings. Within the guidelines described above, the evaluator has 
considerable latitude in the framing of questions for his own design 
meeting. 

(2) . BRIEF DISCUSSION LEADERS 

Ordinarily, the evaluator plays a rather inactive role at the design 
meeting for his program. This frees him to observe any aspect of the 
meeting at any time and enables him to make a better judgment of the 
overall activity than would be possible if he were responsible for a 
definite assignment. 

The person in charge of each discussion group is the discussion 
leader, who is usually an evaluator of another program or some other 
member of the evaluation staff, 

A thorough briefing of the discussion leaders by the evaluator will 
help assure a successful design meeting. At the briefing session the 
evaluator should provide the following: 

(1) Enough background inform.ation about the program to provide an 
adequate frame of reference for guiding the discussion groups 




(E) An explanation of the strategy and procedures for the meetings, 
including provisions for recording the proceedings 

(3) A description of the composition of the discussion groups and 
identification of key program personnel 

(4) An examination of the purpose of each question to be asked and 
suggestions for ways of obtaining the information desired 

(5) A discussion of tlie role of the discussion leader, including ways 
of obtaining assistance from the evaluator if he is needed during 
the meeting. 

The importance of holding the discussion groups to consideration of 
the predetermined questions should be emphasized at the briefing session. 
Discussion leaders should be cautioned about departing from these ques- 
tions for any extended period, since experience has shown that time does 
not permit digression at the desigrx meeting. The importance of pacing 
should also be discussed, and a suggested time schedule be agreed upon. 

Often valuable suggestions from experienced discussion leaders at 
the briefing session can help' improve the conducting of the design meet- 
ing. Nevertheless, all decisions concerning the meeting rest with the 
program evaluator, subject to the approval of the Coordinator of Evalua- 
tion. 



2. EVALUATION OF AN ONGOING PROGRAM 

Whenever a new evaluator is assigned to an ongoing program, it is 
highly desirable that he become promptly identified with it in the minds 



t 



oi program staff. It is also essential that the moinentum of evaluation 
be maintained during the transition period between evaluators. To these 
ends, ‘the Coordinator of Evaluation will acquaint the evaluator with the 
history of the program and will outline the projected strategy and time- 
line for evaluation for at least the next several months. 

I 

a. CONFER WITH PREVIOUS EVALUATOR OF THE PROGRAM 

« 

An informal meeting with the previous evaluator of the program 
should be arranged if he is available. In many ways, he is the person 
best able to bring about a smooth transition in the evaluation of the pro- 
gram, He can, for instance, alert the new evaluator to the most effec- 

« 

tive ways of interacting with program personnel at several levels of 
responsibility. 

If the previous evaluator is a current employee of the Office of 
Research, the Coordinator of Evaluation may decide to have him work 
with his successor on a part-time basis during a brief break-in period. 
This time can be spent examing the files, planning forthcoming activities 
and meeting jointly with the program manager, with the new evaluator 
gradually taking over the feigns from his predecessor. 

b. BECOME FAMILIAR WITH PROGRAM FILES 

As previously stated, the librarian maintains a file of evaluation and 
related activities for all ongoing programs. The new evaluator should 
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read these files carefvdly, take notes, and discuss them with the pre- 
■vious evaluator as appropriate. 

As pointed out in section b, page 11, the previous evaluator will piob- 
ably have his own confidential files for the program. These may contain 
memoranda and notes to supplement the official formal files. At his 
discretion they may be made available to the new evaluator, who can 
generally gain much useful information from them. 

The new evaluator will, of course, assume the responsibility for 
keeping the official program files current. He is encouraged to set up 
his own informal program file to suit his own needs. His attention is 
again called to section b above. 

* 

c. READ RELEVANT LITERATURE AND CONTINUE FILE OF 
PERTINENT INFORMATION 

As he reads the files for his program, the new evaluator will likely 
discover references to a body of literature concerning his program. 

This should be skimmed and he should form the habit of keeping abreast 
of new books, articles, and reports that pertain to the program. Atten- 
tion is called to section a, page 10, for further observations on this 
subject. 

do PREPARE CYCLE PLAN AND TIMELHsE FOR STAGE I 
ACTIVITY 

It is possible that the new evaluator’s ongoing program will not yet 
have achieved a stabilized design when he assumes responsibility for it. 
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This will mean that further Stage I activity is indicated.. In this case, 
the evaluator should prepare a cycle plan and timeline which will center 
on improving the program design. It will pay to examine the existing 
design quite critically. If a panel has met to judge the adequacy of the 
desigh, its observations and recommendations should also be closely 
studied. This research will identify the typo of Stage I activity that is 
needed, and the point at which’ it should proceed. 

In the event that additional Stage I work is not required or not justi- 
fied at this time, the proper point for evaluation of the program should 
be determined, and the evaluator should proceed as specified for that 
stage in the discrepancy model, 

e, MAKE INITIAL CONTACT WITH PROGRAM MANAGER 




^JC 



Except. for a slight difference in emphasis stemming from the pro- 



gram manager *s experience with the evaluation of the program, the 



purposes and procedures for meeting the program manager are the same 



as described in section d, page 12. The initial contact should be made 



as soon as the decision concerning the next cycle of the program* s eval- 



uation has been confirmed. 



(1) DISCUSS OVERALL EVALUATION PROCEDURE WITH 
PROGRAM MANAGER 



The approach used at the initial meeting between the new evaluator 



and the program manager of an ongoing program is essentially the same 



as it is for any new program, (See section(l), page 12,.)' The Coordinator 
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of Evaluation and the previous evaluator, if available, are in the best 
pofviiion to know which points need to be reinforced and which ideas can 
safely be taken for granted at this first meeting. The new evaluator 
should be guided by their advice concerning these matters and the extent 

to which he should participate in the meeting, 

# 

(2) ARRANGE WITH PROGRAM MANAGER TO CONTINUE WITH 
CURRENT CYCLE OF STAGE I EVALUATION IF REQUIRED 

During the first meeting between evaluator and program manager, 

an agreement will ordinarily be reached to continue with Stage I evalua- 

f 

tion if such activity is desirable. The evaluator will then proceed as 
the logic of the situation requires in accordance with the basic plan of 
the Discrepancy Evaluation Model. 



B. GENERATING THE PROGRAM DESIGN 
1. HOLD DESIGN MEETING 

On the day before the design meeting, the evaluator should make a 
final check to see that all routine details, from identification badges 
to the arrangement of furniture, are in order. 

A typical program design meeting lasts several hours. It begins 
with all participants assembling in a large group, where they are wel- 
comed by a high-ranking member of the program staff. The Director 
of Research then offers a brief explanation of the purpose and rationale 
for the program design meeting. The Coordinator of Evaluation 
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• introduces the program evaluator. Discussion leaders are identified, 
and participants are directed to their assigned groups. These intro- 
ductory activities should take no longer than fifteen or twenty minutes. 

The heart of the design meeting is the actual deliberation of the 
discussion groups. Here the careful planning will be evident in the 

* M 

animated and productive proceedings conducted imder the unobtrusive 
control of well-prepared discussion leaders. In each group, a qualified 
recorder will be noting in detail all answers to the discussion leader’s 
questions. The record of each discussion group will be given to the 
evaluator at the end of the meeting, 

A good way to close the program design meeting is to have the 
participants reassemble in a large group for five to ten minutes. The 
Coordinator of Evaluation 'will thank the program staff for its contri- 
butions and discuss the general plans for disseminating the design which 
the discussion groups have generated. These closing minutes offer an 
excellent opportunity to reinforce the idea that the evaluation staff exists 
to perform an important but non-thr eatening role in producing effective 
programs in the Pittsburgh Public Schools, 

2. EVALUATE DESIGN MEETING 

The program design meeting as it is presently perceived has evolved 
largely through the candid evaluation of previous meetings. The evalua- 
tion session should follow shortly after the design meeting while details 




t 



and impressions are fresh in the participant's minds. All professional 
members of the evaluation staff who participated in the design meeting 
should attend the session. It is quite likely that the discussion at any 
follow-up meeting will produce several valuable suggestions for modifi- 
cations in the basic procedures for obtaining a program design. 

3. CONSTRUCT THE PROGRAM DESIGN 



The records of the proceedings of each discussion group contain 
the content for the program design. It is the evaluator's task to extract 
the areas of consensus from these records for each program variable 
and to fit them into the design format reproduced in Figure 3 on page 17 

f 

of the Discrepancy Evaluation Model. Outcomes should be phrased in 
behavioral terms where possible. 



Where the records fail to indicate consensus, the evaluator must be 



careful not to force a spurious agreement. The wiser course is to list 



the variables about which consensus is lacking. Those considered to be 



clearly routine may be resolved by a conference with the program manag- 
er. Those which are essential to the program's input or operation, or 



which are crucial to realization of the stated outcomes should be listed 



as sources of discrepancy in the preliminary design. A decision concern- 



ing them can then be made by the panel which will be convened to judge 



the adequacy of the program's design. 
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4. SUBMIT DRAFT OF PROGRAM DESIGN FOR APPROVA.L AND 
EDITING 

When the draft of the program design is finished, it should be pre- 
sented to the Coordinator of Evaluation for revision or approval. It 
will then go to the editor for final processing and duplication. The eval- 
uator and the editor will meet informally whenever necessary during the 





editing process to discuss matters relating to the final form of the pro- 
gram design. 

5. DISTRIBUTE THE PROGRAM DESIGN 

Prompt feedback to program staff at all levels is a cardinal feature 
of evaluation under the discrepancy model. Therefore, as soon as the 
program design has been cleared for publication, it should be duplicated 
in sufficient quantity for distribution to the program staff. The distribu- 
tion list should be decided in conference with the Coordinator of Evalua- 
tion. It will be remembered that all participants at the design meeting 
were promised punctual feedback. The best feedback stemming from a 
design meeting is the program design itself. Since the design will proba- 
bly need revision, however, the evaluator should prepare a covering let- 
ter, to be signed by the Coordinator of Evaluation or the Director of 
Research, which clearly indicates the interim nature of the document. 




t 



C. AMENDING THE PROGRAM DESIGN 



1. GENERAL PURPOSE AND DESCRIPTION OF THE PANEL 
MEETING 



Once a program design has been obtained, questions concerning its 



adequacy arise. These questions must be answered before the design 



can stand as an acceptable standard of measurement. Specifically, the 
design will be judged "with respect to its compr ehensiveness ^ its exter- 



nal validity, and its internal consistency, Compr ehensiveness refers to 



the completeness of the design- -Is the definition of the program precise 
in all its elements, or are there significant gaps that need to be filled 



in or criteria that need to be specified? External validity concerns the 
basic compatibility of a program with the total operation of the school or 
other educational agency in which it functions. It is also concerned with 



the program s degree of harmony with the philosophy and objectives of 
the school system. Internal consistency (or internal validity) relates 
to whether a program is consistent with itself- -Are its component parts 
consonant with one another, and do they all contribute to the realization 



of the stated outcomes? 



Judgments concerning these three major areas will be made by a 
panel which will examine the program design item by item. The typical 
panel is made up of the program’s director and an assistant, the Co- 



ordinator of E^luation and the evaluator, and a neutral consultant with 



expert qualifications in the discipline to which the program relates. The 
Coordinator of Evaluation (or in his absence the program evaluator) will 
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preside at the panel meeting. It is his task, where possible, to lead the 



panel to ratify each item in the design^ either as it stands or as discus- 



sion and argument caus^ it to be amended. It must be emphasized that 



this task does not permit the evaluation staff to determine the content 



of the design or to force consensus where none exists. Unresolved issues 
remaining after the panel meeting should be identified as such and ear- 



marked for reconsideration and possible modification at a later date. It 



should be obvious, of course, that serious design conflicts are likely 



sources of malfunctions in program operation. 



2. OBTAIN FIELD DATA TO TEST PROGRAM COMPATIBILITY 
OF ONGOING PROGRAM 



Despite the inadvisability of installing a program before it has been 



adequately designed, practical necessities often cause such action to be 



taken. In these cases, the evaluator can readily obtain compatibility data 



by field observation prior to the panel meeting. These data should be 



collected and analyzed in time to be reported when the panel considers 



the question of compatibility. In collecting the data, the evaluator should 



follow the steps outlined below. 



a, DETERMINE METHOD OF DATA COLLECTION 



The type and method of data collection for compatibility testing 



depend upon the available time, sample size, program complexity, and 



practical matters such as the time of the school year, administrative 
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directives, and staff availability. Thus, the type and method of data 

collection need to be determined by the evaluator and Coordinator of 

» 

Evaluation with these and other appropriate factors in mind. The two 
most common approaches are the field interview and the questionnaire 
response, each of which requires a custom-made instrument to fit the 
program being studied. 



b. DESIGN INSTRUMENT FOR COMPATIBILITY DATA COLLECTION 



The general instrument file as well as the files for specific programs 



contain examples of instruments used in compatibility testing. Often one 



of these can be adapted to fit the needs of another program. Each ques- 



tion or other item in the instrument should be justified by a written ratio- 



nale. Although the number of questions asked, the phrasing of the items. 



and their arrangement will vary with the program, answers to such com- 



patibility questions as the following can be obtained either through inter- 



views or questionnaires: 



1, What do participants give up in order to take part in this pro- 



gram? 



2, What arrangements , if any, are made for participants to make 



up the work they miss in order to take part in the program? 



3. What do teachers or other staff give up in order to make this 



program possible? 



4. Does this program infringe on any other program in any way? 



If so, which program(s), and in what way(s) ? 
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5, How does this program help the school improve its total re- 
sponsibility to the student and community? 

6, What effect has this program had to date on the attitudes of 





participants and staff? 

After the instrument has been designed, edited, and approved, it 
will be given to the administrative assistant for duplication in the re- 
quired quantity. As with all other instruments, copies will be placed 
in the files. 

Under ideal conditions, every i.nstrument should be tested and re- 
vised as necessary before it is used to collect data for analysis. It is 
realized that this may not always be possible in the case of compatibility 
instruments because of time constraints, but it should be done whenever 
it is feasible. 



c, TRAIN INTERVIEWERS 

If data are to be obtained through interviews, it will be necessary to 
train the. interviewer s in appropriate techniques. This v/ill include the 
psychological aspects of interviewing as well as the conditions peculiar 
to a given program that may alter the inter viewer *s approach. Care must 
be taken to assure that all interviewers will follow the same general pro- 
cedures, and guidelines should be drawn up to encompass permissable 
variations. An attempt should be made to establish interrater reliability 
among interviewers, A practical discussion of interviewing is found on 
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pages 23-47 of Evaluation Models and Manual; Title III ESEA Educational 



Progi^ams (Now Jersey Department of Public Instruction), a copy of which 
is in the library. 

d. CONDUCT FIELD TESTING OF PROGRAM COMPATIBILITY 

The administrative assistant will arrange for collecting data as soon 

as strategy and procedures have been approved. Prior approval is a 

prerequisite to any data collection in the Pittsburgh Public Schools. It 

is essential that appointments be kept as made and necessary modifications 

be arranged well in advance, so as not to interrupt school operations. A 

source of conflict between school or administrative staff and the evaluation 

office concerning data collection should be referred immediately to the 

Coordinator of Evaluation or the Director of Research, who should also 

be notified when emergencies arise. The evaluator will discover that 

close attention to established routines for data collection will facilitate 

his task and create a good working relationship between field personnel 

and the evaluation staff for his own and other programs, 

♦ 

It is a good idea to keep a running tabulation of day-to-day findings 
when collecting field data and to compare notes regularly with interviewers. 
Experiences in the field, especially in the early days of the data collection, 
may lead to desirable procedural changes and instrument modification. 



3. ANALYZE FIELD DATA AND RECORD FINDINGS 

When compatibility testing has been completed, the results will be 
analyzed to establish findings and reach general conclusions. The nature 
of the analysis vSupports the purpose of the compatibility testing, v^^hich is 
to discover whether the program has an overall external consistency in 
the opinion of those who are most intimately concerned with it. Results 
for each item or question should be reported separately, with specific 
statistics available in case they are requested at the panel meeting. 

4. PLAN PANEL MEETING TO ASSESS PROGRAM DESIGN 

Plans for the panel meeting to judge the program design may be 

Emulated before compatibility testing has been completed. The rules 
for planning the design meeting as discussed in section d, page 5, gen- 
.erally apply, except that the procedure is simplified because of the 
smaller size of the group. Specific tasks are outlined below. 

a. IDENTIFY CONSULTANT 

The consultant is a key member of the panel convened to assess the 
program design. He is a high-status figure, chosen both for his insight 
into the theory underlying the program and his competence to evaluate 
its educational application. His qualifications rnust be acceptable to 
•program and evaluation staff alike if his recommendations are to be taken 
seriously. Since he will have no vested interest in the program, he can 
act as an objective appraiser of the program design. Often he will be 



employed at a local university or research organization! The Director 
of Research, the Coordinator of Evaluation, and in some cases the 
program manager will help identify the consultant. Approval rests with 
the Director of Research. Contractual arrangements are handled by 
the Coordinator of Operations.. 

b. ORIENT/CONSULTANT TO PROGRAM 

As soon as a consultant has been engaged, the evaluator will arrange 
to meet with him to provide necessary background information about the 
program he will be asked to judge. At this meeting special emphasis 
should be given to the consultant's role at the panel meeting. At this time 
he shoi.ild be given a copy of the design and other pertinent documents, A 
brief interpretation of the discrepancy evaluation rationale will also be in 
order. 

c. ACCOMPANY CONSULTANT ON FIELD OBSERVATION 

The consultant should be given the opportunity to make an on-site 

observation of the program. This will provide him with a sharper frame 
of reference in which to make his judgments. It may be possible for the 
evaluator to combine the consultant's visit with a compatibility interview. 

5. HOLD PANEL MEETING 

The personnel and characteristics of a program will influence .the 
style of the panel meeting, but its general purpose is the same for all 
programs, A full treatment of the panel meeting will be found on 
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pages 18-23 of the Discrepancy Evaluation Model . This, document also 
summarizes the activity of an actual panel meeting as part of the case 
history of the Standard Speech Development Program (pages 51-52). 
Listening to the tapes of previous panel proceedings will alert the eval- 
uator *to the interaction, pacing, and overall "feel" of a panel meeting, 

a. ASSESS EACH ELEMENT OF THE PROGRAM DESl'GN 

In his opening remarks, the Coordinator of Evaluation will introduce 
the members of the panel, state their qualifications, and describe the 
purpose and procedures of the meeting. He will then direct the panel to 
the program design which will be discussed item by item. A stenographer 
will record the proceedings. Experience has shown that panel members 
readily accept the taping of the entire proceedings. This tape will prove 
invaluable in preparing the report of the panel meeting. It can serve as 
an unbiased source for refreshing memories of program and evaluation 
staff about the decisions which were reached for amending the design. 



b, REPORT COMPATIBILITY FINDINGS 



Early in the meeting, the evaluator will report the findings obtained 
from the analysis of compatibility testing in the field. He should describe 



the method used to gather data, state the questions asked, and summarize 



the answers. He should be prepared to report specific totals and percen- 



tages if requested, but since an overall impression of program compati- 
bility is the purpose of his report, his remarks should be general. He 
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should conclude his report by indicating areas of consensus and disagree- 
ment on the program's compatibility as seen in the field. During the 
remainder of the meeting, he will answer questions and offer feedback 
concerning evaluation of the program and otlierwise contribute to the dis- 
cussion as circumstances suggest, 

6. PREPARE RECORD OF PANEL PROCEEDINGS 

The record of panel proceedings reproduces the original program 
design and adds all recommended modifica-tions in their proper sequence. 
An example of a record of panel proceedings appears as Appendix B 
of the Discrepancy Evaluation Model. 

a, DISCUSS RECORD OF PANEL PROCEEDINGS WITH PROGRAM 
MANAGER 

After the record of panel proceedings has been edited but before its 
final publication, the evaluator should discuss it with the program mana- 
ger. Minor changes in emphasis or interpretation may legitimately result 
from this conference, but all of these should be supported by the record. 

A copy of the completed record of panel proceedings shovild be given to 
the program manager, but it is not ordinarily distributed to the entire 
program staff, 

7. WRITE AMENDED PROGRAM DESIGN 

The evaluator will now prepare a revised program design, incor- 
„ porating the changes recommended by the panel. Although the written 
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transcription and the tape of the entire proceedings should provide an 
accurate record of the modifications suggested, the evaluator is free 
to request clarification on any point from any participant at the meeting. 

The revised design will be distributed to all people associated with 
the program. It may well become the subject of a cycle report and should 
be included in all major evaluation documents of the program. 

8. USE AMENDED DESIGN FOR STAGE II EVALUATION 

The program is now ready for Stage II evaluation which seeks to 
compare design with actual operation. The strategy of using program 
staff in design activity and the concept of continuous feedback are implicit 
in'the above material. The revised design should therefore be a clearly 
visible standard for further evaluation of the. program. 
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Section III 

STAGE II EVALUATION 

A. PLAN STAGE II EVALUATION 

Stage II evaluation can begin as soon as the first amended design 
has been distributed to the program staff. Even tho\igh the design may 
requii-e further work, it is usually possible and desirable to measure 
some aspects of the program's operation. The standard for comparing 
administrative expectation with field reality is the amended design which 
incorporates the modifications approved at the panel meeting. 

Stage II. evaluation can never be a haphazard exercise. It must 

0 

examine systematically and thoroughly the degree to which the separate 
elements defined in the design are being put into practice. There are 
several sources to which the evaluator can turn in determining which 
Stage II variables to include in a cycle of evaluation: 

1. The input and process variables as they appear in the design 
(These should be randomly sampled, ) 

2. The areas of most frequent or most serious disagreement as 
stated by field staff at the design meeting or in compatibility 
testing 

3. Items recorded in a problems inventory obtained at the close 
of the design meeting 

4. Elements in the design which may have been identified for 
further clarification at the panel meeting 



- 34 - 












I* ^ 



Two or more of these sources can be combined in preparing the 



list of variables for Stage II observation. It is also possible to include 



questions pertaining to other stages of evaluation in a cycle plan, since 



a program is rarely confined to a single stage of development at any 



given tim,e. 



1. PREPARE CHART OF CYCLE PLAN 




Plans for an evaluation cycle can be charted as shown in Figure 1. 



In preparing this chart, the evaluator will simultaneously organize his 



thinking and illuminate his specific tasks in planning and conducting his 



observations. Together with the accompanying timeline discussed below. 



the chart will constitute a concise record of related evaluation activities. 



The events listed in Figure 1 are merely a sample of the many variables 



that might be tested in a single cycle. Complete charts for cycles of 



various programs are found in the files. 



2. PREPARE TIMELINE FOR CYCLE PLAN 



Figure 2 shows a typical timeline to accompany the chart of the cycle 



plan. In setting it up, the evaluator should keep in mind the importance 



of timely feedback to program staff. He should consult the librarian for 



appropriate literature on PERT charting or other systems of network 



manaigement. These techniques can be a valuable aid in constructing a 



realistic chronology. If applied to cycle planning, they will increase the 
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likelihood of the smooth progression of evaluation commitments and 
enable the evaluator to meet projected deadlines, 

3. DESIGN NECESSARY INSTRUMENTS 

The evaluator should refer to section f, page 6, above for a 
discussion of Stage II instrumentation. The ideal instrument will be as 
brief and straightforward as possible. It will seek to collect only such 
data as are needed to fulfill the intent of the cycle plan. It will permit 
observers to record information with minimum interference in program 
activities. It will be amenable to high inter -rater reliability and will 
have low risk of inaccurate interpretation. If data processing is indi- 
cated, it will be compatible with computer requirements. Finally, it 
will be constructed so that the data it yields can readily be subjected to 
.appropriate statistical procedures. 

In aiming for the ideal Stage II instrument, the evaluator will con- 
fer with various staff members of the Office of Research, whose skills 
are at his disposal. The Coordinator of Evaluation will arrange for 
consultant services when necessary. The evaluator should allow suffi- 
cient time for instrument design so that it does not cause delay in con- 
ducting field observations, 

4. FIELD TEST INSTRUMENTS AND TRAIN RATERS 

A field test of the instrument should be conducted in order to deter- 
mine its general suitability and to identify any areas or items in need of 
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clarification. Extensive field testing is usually not required. It should 



terminate as soon as the evaluator is satisfied that raters and respon- 



dents are able to cope with the instrument, and that it is supplying the 



kinds of data it was intended to produce. When the field testing is com- 



pleted, necessary revisions in the instrument should be made immediately 
so that actual data collection can begin. If major changes are called for. 



or if a completely new instrument is required, the design and field test- 



ing procedures desjcribed above will have to be repeated. 

— . The evaluator will formulate a set of precise directions for admin- 



istering his instrument. The directions should be clearly worded in 
non-technical language. Sample responses should be included where 



necessary. Specific instructions concerning timing, environmental con- 



ditions, methods of recording responses, and other administrative con- 



siderations should be included in the directions where appropriate. 



Figure 3 and 4 reproduce samples of a Stage II instrument and the direc- 



tions for administering it. 



The Coordinator of Evaluation will advise the evaluator where the 



field testing of the instrunaent will be administered. The administrative 
assistant will set up and confirm the testing schedule. Preceding this. 



it will be necessary to assign and train raters if the evaluator will not 



be collecting the data or if he will need assistance. The training process 
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^should continue into the instrument testing period so that riaters and eval- 



uator can improve data collection procedures together in the light of 



conditions they encounter in the field. 
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TRANSITION ROOM 

DIRECTIONS FOR ADMINISTERING CYCLE II 
OBSERVATION SCHEDULE 



Preliminary Directions 

Before beginning the actual observation, the observer jvill record 

t 

the following information at the top of each page; 

1. School 

2. Teacher 

3. Observer 

4. Visitation Number 

5. Date (including day of week, month, and day of month) 

6. Grade 

7. Number of students present 
General Information 

1. This observation schedule is designed to enable the observer 
to focus his attention on the teacher*s activity and his use of instruc- 
tional media, 

2« Observations will be recorded precisely at 3-minute intervals, 

3, Each observation session will normally include six separate 
observation, periods, 

4, The exact time (hour and minute) for each observation period 
should be recorded in the space below the number. 

5, The observer will register all pertinent items by recording a 
single stroke (/) in the appropriate block or blocks. 

f 

6, After all the observation periods have been recorded, the total 
number of strokes should be entered in the "Total" column at the extreme 
right of the page. 
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7. It is contemplated that each teacher will be obseiwed on three 
separate occasions during this cycle. In order to insure a complete 
description of the activities and instructional media being used in 
transition rooms, it is important that no two observation sessions for 
any particular room take place on the same day of the week or at the 
same time of day, 

' * 

Activity Categories (Parts I and II) 

' 1, The observer will record each activity with a stroke (/) 

in the appropriate block or blocks for each 3 -minute observation. 

2. If a teacher is engaged in more than one activity at the 
moment of observation, place a mark in every appropriate block. 

♦ 

3. If the teacher's activity involves the whole class, the mark 
should be made in the appropriate block under Part I (Group Instruc- 
tion). If he is engaged with a sub-group within the class or is working 
with an individual student, the mark should be made in the appropriate 
block under Part II (Individual Instruction. ) 

Instructional Materials Used (Part III) 

# 

1. The recorder will record a stroke in each appropriate block 

to indicate all materials and equipment being used at the time of obser 
vation. : 

2. If the teacher is using an instructional medium not listed on 
the observation schedule, briefly indicate the activity and medium on 
the reverse side of the form. 



















FIGURE 



School^ 

Dale 



Teacher 



Grade 



• (day of wk. » md. , day of mo. ) 

« 

Activity Categories 

i 

I. Group Inat ruction (whole class) 



Stage II Instrument 

• O bserve r 

^No. of Students 



i/68 



Visit No. 



1 . 

2 . 

3. 

4. 

5. 

6 . 



10 . 

u. 

12 . 

13. 

14. 

15. 

16 . 

17. 

18. 

19. 

20 . 
21 . 



Lectures 
Demonstrates 
Asks questions 
Reads to class 
Explains 

Checks pupils' work 
Gives answers 

responses 
responses 



7. 

8 . Accepts pupils' 
9« . Rejects pupils' 



Corrects pupils' responses 

Listens to recitation or reading 

Listens to pupils' spontaneous conversation 

Directs group instructional activities ^ 

Directs non-instructional activities 

Distributes or collects instructional material 

Tests pupils 



Corrects pupils' behavior 
Walks around the totin' 

Watches pupils work 

Other miscellaneous teacher activities 



n. Individual Instruction (sub-groups or individual) 



1 . 

. 2 . 

3. 

4. 

5. 

6 . 

7. 

8 . 
9. 

10 . 

11 . 

12 . 

13. 

14. 

15 . 

16 . 

17. 

18. 
19. 
20 * 
21 . 
22 . 



Lectures 

Demonstrates or explains 
Asks instructional questions 
Asks non-instructional questions 
Reads to sub-group 
Checks pupil's work 
Gives answers 
Accepts pupil's responses 
Rejects pupil's responses. 

Co-rrects pupil's responses 
Listens to recitation or reading 
Listens to. pupil's spontaneous conversation 
Directs or suggests instructional activities. 
Directs non-group instructional activities 
Distributes or collects instructional material 
Tests pupils 

Praises pupil's responses or behavior 
Criticizes pupil's responses or behavior 
Discusses pupil's progress 
Discusses pupil's learning difficulties 
Assists pupil with his work 
'Other miscellaneous teachei* activities 



ni. Instructional Material Used 



1. 

2 . 

3. 

4. 

5. 

6 . 

7. 

8 . 
9. 

10 . 

11. 

12 . 

13. 

14. 

15. 

16 . 

17. 

18. 
19. 



Basic readers 

Basic social studies text books 
Reference books 
Other books 
Workbooks 

Duplicated worksheets 

Compositions 

Blackboard 

Flashcards 

Notebooks 

Puzzles 

Charts 

Filmstrips 

Filmstrips and records 

Phonograph records 

Magazines or similar publicationi? 

T-Flasher 

Instructional Games 

Other instructional materials 
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5. ESTABLISH INTER-RATER RELIABILITY 



If the instrument is to be administered by more than one rater, it 



will be necessary to establish inter -rater reliability for it. This can be 



done as part of the testing process by having all raters record the data 
obtained from the same respondents and then comparing the responses 



from each item. Of course, if the instrument is significantly modified 



asl a result of field testing, a new measure of reliability will be required 



Statistical procedures concerning inter -rater reliability should be dis- 



cussed with experts in measurement in the Office of Research. 



6. ADMmiSTER STAGE II INSTRUMENT 

« 

The Administrative Assistant will arrange for raters to collect data 



as soon as the instrument has been modified as the result of field testing 

‘ t 

and inter-rater reliability has been established. The procedures for 



administering the instrument will ordinarily be the same as those followed 



in the field testing period. Frequent reference should be made to the 



written directions mentioned above. Modifications in administering the 



instrument must h ive the approval of the Coordinator of Evaluation. 



7. ANALYZE DATA 



Data analysis shouJ,d be initiated as soon as field observations have 
been completed. The evaluator will recall that the rationale section of 



the cycle plan includes a statement concerning the analysis contemplated 



for eVery area of the design scheduled for observation. At this point it 
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it should be a simple matter to subject the data to the analytical pro- 
cedures already planned. The results of the analysis should be re- 



duced to a series of findings, conclusions drawn according to approved 
research techniques, and preparations made for reporting the findings 



and conclusions to program staff. 



8. WRITE CYCLE REPORT 



The cycle report noting the findings and conclusions obtained from 



the data analysis of Stage II field observations is a brief document which 



attempts to point out to program staff the observed differences between 



design requirements and actual practice. All the variables studied 



should be reported. Where discrepancies are found, alternative ways 

of correcting them should be discussed. The options, however, should 

♦ 

not be^presented as recommendations, nor should the evaluator express 
any preference among them, since changes in design or operation of a 



program are always the prerogatives of program staff. 



As pointed out in the orientation sessions devoted to human relation: 



(See section 3, page 7), application of the Discrepancy Ev aluation 
Model will more often than not lead to the reporting of negative findings. 
In the interest of providing data to correct deficiencies and spark im- 



provement in a program, these findings will be highlighted in the cycle 
report. But they should not be exaggerated. Nor should positive find- 
ings be minimized, since program staffs have a right to know in what 



respects their programs are fimctioning properly. 



-45- 









mrnrnrnimmmmmsmmmmmmmmmmmsmmm 



Cycle reports, while adhering to a standardized approach, can vary 
to reflect the needs of particular programs and the writing styles of 
individual evaluators. They should avoid circumlocutions and unneces- 
sarily technical expressions because they are intended for a large audi- 
ence with varying degrees of sophistication in research concepts and 
terminology. In any case, a report written in uncluttered English is more 
•likely to produce results than one phrased in esoteric language. 

An example of a cycle report dealing with Stage II concerns appears 
as Appendix A of the present document. 

9. DISTRIBUTE CYCLE REPORT 

A distribution list for cycle reports should be determined by con- 
sulting with the Coordinator of Evaluation and the program manager. In 

t 

A 

keeping with the principle of frequent feedback to everyone with a legiti- 
mate interest in the program, the evaluator should work for the broadest 
circulation possible for the cycle report. A covering letter which ex- 
plains the purpose of the report, signed by the Director of Research, 
should be attached to every copy. This letter may solicit comments, and 
reactions from recipients of the report. A sample letter appears as 
Appendix B of this document. 
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CYCLE II REPORT 
KINDERGARTEN PROGRAM 



Evaluation of the Kindergarten Program has been limited to ob- 
taining a program design and informal review of that design. Lack of 
specificity in the design as revised during the 1967-1968 school year 

made necessary further work in this area in 1968-1 969 . This design 

• # » ® 

work included a task force meeting of experienced teachers to discuss 
the functions of the teacher, discussions of the design with kindergarten 
supervisors, and work on objectives using the courses of study for the 
kindergarten as a source of information. 

In addition, compatibility studies were initiated in 1 968 - 1969. The 

tt ’ 

pfesent report is a summary of those studies. 



Compatibility of the Program 

No significant conflicts between the objectives of the Kindergarten 
Program and those of the rest of the school system have become ap- 
parent. However, adequacy of materials, supplies, and facilities is 
questioned by many teachers and some principals. 

A questionnaire concerning adequacy of materials, supplies, and 
facilities was distributed to 30 kindergarten teachers in December of 
1968 . Twenty- nine teachers returned the questionnaire. Of these, 14 
were from non-compensatory schools, and the remaining 15 were from 

I . 

compensatory schools. 
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Teachers were asked whether they had an adequate supply of 
materials in each of several categories. The category in which the 
most "yes” answers were received was that of art supplies. In des- 

t 

cending order of frequency of "yes" responses, the answers were as 



follows:- 



Category 
Art supplies 

Books and pictures • 

Toys for dramatic play 

Toys for gross motor play 

Musical equipment 

Measuring tools 

Materials for science activities 

Toys designed to develop manipulative skills 



% Yes Responses 

-i 90% 

69 % 

62 % 

59% 

59% 

. :• 41% 

f 

• 38 % 

38% 



•No significant difference between compensatory and non-compensatory 
schools with respect to adequacy of any of these supplies appeared 

(p=. 05). 

Teachers were also asked about the adequacy of facilities. All 
teachers reported access to sinks with running water, small tables and 
chairs, lavatory, storage space (often termed inadequate). One teacher 
reported no access to audio-visual materials, two reported insufficient 
indoor play space, two reported insufficient display space, seven 
reported no access to outdoor play space, and nine reported no access 
to cooking equipment. Audio-visual materials were shared half the 



• , V 
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time, and outdoor play space was shared in 9 of the 22 instances in v/hich 
it was availa.ble. Other facilities were occasionally shared, most often 
the lavatory. 

Half the teachers felt that sharing facilities created scheduling or 

other problems. Eleven of the 15 compensatory school teachers had 
*. • 

aides, five of whom were shared with other feachers. 

Teachers were given an opportunity to discuss problems of the 

> 

Kindergarten Program at the end of the questionnaire. Frequent com- 
ments were made about the large size of classes and the sharing of 
classroom space with other programs. Teachers also complained of 

lack of adequate preparation time. Suggestions for alleviating this 

. # 

problem included introducing an aide into the classroom, even once a 
week, and use of volunteers. 

In January tliis questionnaire was follow'ed by two other question- 
naires, one for teachers who had not answered the first questionnaire 
and one for elementary school principals. The questionnaire sent to 
principals was designed to investigate the progression from preprimary 
through kindergarten to first grade, or from kindergarten to first grade 
in those schools where there is no Preprimary Program, The sharing 
of facilities was also investigated. 

Twenty- nine of the 30 questionnaires sent out were returned. 

Results are summarized below (not all principals answered all questions). 
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1. In your opinion, is there a natural progression 

in curriculum and teaching methods from the 
Preprimary Program to the Kindergarten 
Program? • 

From kindergarten to first grade? 

2. Was a room intended for kindergarten use 
built into your school? 

Does kindergarten occupy that room? 

3. Is the room kindergarten occupies big 
• enough for present enrollment? 

4. Does kindergarten share its room with 
any other program or other school function 
(lunch room, auditorium, etc. )? 

5. Is outdoor play space available to kindergarten? 

6. Must kindergarten share any of the following 
facilities with other school groups? 

.. Audio-visual materials 
♦ 

Lavatory 

Outdoor play space 
Indoor play equipment 

If yes, do any of these shared facilities cause 
scheduling conflicts when they are used? 



YES 


NO 


7 


. 


' 22 


7 


27 


2 


27 


0 




\ 

8 


7 


21 


, 2a 


1 


. IS 


10 


. ^2 


15 


17 


10 


5 


21 


7 


19 



No significant difference in distribution of replies between non-, 
coinpensator y and compensatory schools was determined (p=. 05). 

Twelve of the principals replying took the opportunity to make com- 
ments about the program. Of these, three commented that reading 



schools.. 



ju - ^ 

'TThis question was not asked of principals in non-compensatory 
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I’eadiness programs in the kindergarten should be intehsified. Ten 

commented on lack of space, small outdoor play area, or lack of 

• \ 

* «' ‘ 

materials! Usually more than .one of these problems was mentioned-- 

• A 

lack of space most often. . • 

The second questionnaire sent to kindergarten teachers was confined 

to schools in which there is a Preprimary Program, and dealt chiefly 

with the effect of the Preprimary Program upon entering kindergarten 

students. Communication between the teacher sAvas asked about, as 

was communication between kindergarten and first grade teachers. 

Twenty-three questionnaires were sent out, and 23 returned. This 

does not represent one hundred percent response: only one teacher in 

each school received a questionnaire; however, three principals insisted 

that both teachers fill it out. 

, < 

The average size of classes among the responding teachers was 
34. 5 students, 10. 8 or 31. 4% of whom were former preprimary students. 
Almost all of the teachers knew which of their students had attended 
•preprimary. Only one teacher saw no appreciable difference between 
children who had attended preprimary and children who had not. No 
negative comments were made, and most of the teachers had had some 
communication with both preprimary and first grade teachers. Five of 
*the teachers reported criticism from the first grade teacher about the 
Kindergarten Program. 
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In summary, shortages of materials, cramped facilities, and 
problems resulting from the sharing of facilities and large classes 
exist and are seen by teachers and principals as detrimental to the 
program. The program as a program , however, is praised by both 
teachers and principals. 



* * 
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THE BOARD OF PUBUC EDUCATION 

Pittsburgh, F>a. 15213 







% 







administration building 
OELLCriELD AND FORBES AVENUES 



Dear (Name of program manager): 

In keeping with our policy of providing information on 
program progress, the Office of Research is issuing the attached 
Cycle Report to staff members of the Kindergarten Program. 



The Cycle Report is the result of research conducted 
in the current cycle of evaluation. We hope that you and your 
staff will find it useful in the continued operation of your program. 

Sincerely yours, 



Director of Research 
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